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Tutorial  Outline 

Section  I  -  Understanding  Data 

•  How  to  use  data 

•  Understanding  variation 

•  Requirements  for  success 

•  Common  risks  and  pitfalls 

Section  II  -  Data  Analysis  Dynamics 

•  Learning  from  our  experiences 

•  Useful  tips  for  making  measurement  work 

•  Thread  together  methods,  tools,  processes 
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Tutorial  Outline 

Section  III  -  Case  Study 

Summary 

Addenda 

•  Additional  vignettes 

•  Tool  tips 
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Tutorial  Focus 

Tools,  tips,  and  techniques  your  organization  can  use  for 
analyzing  software  data  and  taking  action 

Specifically  we  will  focus  on 

•  day-to-day  practices 

•  activities  that  lead  to  breakthroughs 

•  why  the  problem,  not  management,  should  drive  your 
measurement  program 

Remember: 

There  is  no  “cookie-cutter”  approach  to  doing  good 
measurement,  but  there  are  some  best  practices. 
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Section  I:  Understanding  Data 

Data  can  help  you 

•  Identify  root  causes  of  variability,  off-target  performance 

•  Better  predict  plans  and  commitments 

•  Make  better  decisions  and  take  action 

•  Monitor  activities  to  keep  projects  on  cost,  schedule 


Data  is  the  means  to  an  end,  not  the  end  itself. 
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Process  Performance  Data 

Data  allows  you  to  access/analyze  process  performance. 

Process  performance  is  behavior  that  can  be  described  or 
measured  using  attributes  of 

•  process  operation  or  execution 

•  resultant  products  or  services 

Process  performance  measures  answer  the  question: 
“How  is  the  process  performing  with  respect  to 
quality,  quantity,  effort  (cost)  and  time?” 

All  process  behavior  is  variable. 
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Getting  at  the  Cause  of  Variation 

Shewhart  divides  variation  into  two  types: 

•  Common  Cause  Variation 

-  variation  in  process  performance  due  to  normal  or 
inherent  interaction  among  process  components 
(people,  machines,  material,  environment,  and 
methods). 

•  Assignable  Cause  (Special  Cause)  Variation 

-  variation  in  process  performance  due  to  events 
that  are  not  part  of  the  normal  process. 

-  represents  sudden  or  persistent  abnormal  changes 
to  one  or  more  of  the  process  components. 
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Understanding  Variation 

Everything  is  a  process. 

All  processes  have  inherent  variability. 

Data  is  used  to  understand  variation  and  to  drive 
decisions  to  improve  the  processes. 


Data  Spread  due  to 
Common  Cause  Variation 


variation  will  re-establish  itself.) 


[ASQ  00],  [ASA  01] 
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In  Other  Words 


Process  Off  Target 


Target 


Defects 


LSL 


USL 


Center 

Process 


Target 


LSL 


Excessive  Variation 


Tar 

get 

■  Defects 

LSL  USL 

USL 


Reduce 

Spread 
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Measurement  Data  Requires 
Analysis  and  Interpretation 


Data 


Analysis 


Interpretation 


Input 


Transformation 


Output 


Separating  signal  from  noise  requires  rigorous  analysis 
procedures. 

This  allows  quantitatively  based  inferences  to  be  drawn 
to  guide  decisions  and  actions. 
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Data  Analysis  Studies 

Remember  what  you  are  trying  to  accomplish.  There  are 
two  approaches  to  data  analysis  to  consider: 

•  Enumerative 

•  aim  is  descriptive 

•  determines  “How  many?”  -  not  -  “Why  so  many?” 

•  Analytic 

•  aim  is  to  predict  or  improve  product  attributes  and/or 
the  behavior  of  the  process  in  the  future 
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Enumerative  Studies 

An  enumerative  study  answers  questions  such  as: 

•  How  many  defects  were  found  inspecting  product 
code? 

•  How  many  problem  reports  have  customers  filed? 

•  What  percent  of  staff  have  been  trained  in  object- 
oriented  design  methods? 

•  How  large  were  the  last  five  products  we  delivered? 

•  What  were  the  average  sizes  of  our  code-inspection 
teams  last  year? 

•  How  many  staff  hours  were  spent  on  software  rework 
last  month? 
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Analytic  Studies 

Software  engineering  examples  of  analytical  studies 
include: 

•  measuring  software  product  attributes  for  the  purpose 
of  making  changes  to  future  products 

•  evaluating  defect  discovery  profiles  to  identify  focal 
areas  for  process  improvement 

•  predicting  schedules,  costs,  or  operational  reliability 

•  evaluating/comparing  software  tools,  technologies,  or 
methods — for  the  purpose  of  selecting  among  them  for 
future  use 

•  stabilizing  and  improving  software  processes  or  to 
assess  process  capability 
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Enumerative  vs.  Analytic 

Undertake  an  enumerative  study  if:  action  is  to  be  taken 
on  the  subject  based  on  data  that  is  already  collected 

Undertake  an  analytic  study  if:  action  is  to  be  taken  on  the 
process  that  produced  the  data 


Analytic  studies  utilize  statistical  process 
control  tools  to  draw  inferences  about  future 

process  performance. 
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Basic  Data  Analysis  Paradigm 


Problem  and  goal  statement  (Y): 

•  maximum  latent  defects  released 

•  minimum  mean  time  between  failure  in  the  field 

•  time  to  market  improvement  (as  function  of  test  time,  defect  density) 


•  Problem  &  goal 
statements 

•  Define  boundaries 

•  Process  maps 

•  “Management  by  Fact” 


•  Discovery:  paretos,  histograms,  distributions,  c&e 

•  Understanding:  root  cause,  critical  factors 

•  Improvement:  adjust  critical  factors,  redesign 

•  Performance:  on  target,  with  desired  variation 

Y  =  f(defect  profile,  yield) 

=  f(review  rate,  method,  complexity . ) 
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Measure 


Analyze 


Act 


Control 


Tips  for  Good  Measures 


Measures  used  to  characterize  products  or  processes 

•  relate  closely  to  the  issue  under  study 

•  have  high  information  content 

•  pass  a  reality  and  validity  test 

•  permit  easy  and  economical  collection  of  data 

•  permit  consistently  collected,  well-defined  data 

•  show  measurable  variation  as  a  set 

•  have  diagnostic  value 


[Wheeler  92] 


©2003  by  Carnegie  Mellon  University 


Version  1.0 


page  16 


Carnegie  Mellon 

Software  Engineering  Institute 


Measure 


Analyze 


Act 


Control 


Tips  for  Better  Data  Analysis 


Verified:  accuracy  of  format,  type,  range,  completeness, 
and  type 


Valid  and  Reliable:  clear,  consistent  definitions 


Accurate  and  Precise:  precise  counting  method 

Based  on  operational  definitions,  you  should  know 

•  What  does  this  measure  mean? 

•  What  are  the  rules  for  assigning  values? 

•  What  is  the  data  recording  procedure? 
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Measure 


Analyze 


Act 


Control 


Tips  for  Operational  Definitions  1 


Three  criteria  for  creating  operational  definitions 

•  Communication  -  will  others  know  precisely  what 
was  measured,  how  it  was  measured,  and  what  was 
included  or  excluded? 

•  Repeatability  -  could  others,  armed  with  the 
definition,  repeat  the  measurements  and  get 
essentially  the  same  results? 

•  Traceability  -  are  the  origins  of  the  data  identified  in 
terms  of  time,  source,  sequence,  activity,  product, 
status,  environment,  tools  used,  and  collector. 


[Deming] 
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Measure 


Analyze 


Act 


Control 


Tips  for  Operational  Definitions  2 


Operational  definitions  also  help  pinpoint  training  needs 
for  data  collection. 


The  cost  of  data  collection  also  bears  on 

•  When  the  data  will  be  collected 

•  Where  the  data  will  be  collected 

•  Who  will  collect  the  data 
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Measure 


Analyze 


Act 


Control 


Tips  for  Better  Data  Analysis 


Why  should  we  care  about  the  data  details? 

Validity  -  apples  to  apples  comparisons 

Reliability  -  understand  the  impact  of  variation 

Accuracy  -  knowing  that  there  is  a  signal 

Precision  -  level  of  certainty  for  responding  to  the  signal 
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Define 

-> 

Measure 

|  Analyze  ^ 

Act 

> 

Control 

Tips  for  Analyzing  Data 

Critical  inputs 

•  Knowledge  of  product  or  process  being  measured 

•  Driven  by  business/  technical  issues  or  goals 

•  Quality  of  measurement  data 

Critical  aspects  of  the  analysis  process 

•  Acknowledgement  of  and  accounting  for  variation 

•  Appropriate  use  of  analysis  tools  and  techniques 

•  Resources  and  references  (people,  books) 
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Define 

-> 

Measure 

|  Analyze  ^ 

Act 

> 

Control 

Take  the  Data  Deeper 


To  reduce  variation  pursue  three  investigative  directions: 

•  Identify  the  assignable  causes  of  instability  and  take 
steps  to  prevent  the  causes  from  recurring. 

•  If  the  process  is  stable  but  not  capable  (not  meeting 
organizational  or  customer  needs),  then  identify, 
design,  and  implement  necessary  changes  that  will 
make  the  process  capable. 

•  Continually  improve  the  process,  so  that  variability  is 
reduced  and  quality,  cost,  or  cycle  time  are  improved. 
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Tips  for  Finding  and  Correcting 
Assignable  Causes 


No  formula  or  transformation  algorithm  is  applicable.  Just 
like  debugging  software  -  it  requires  good  detective  work. 


•  thorough  knowledge  of  the  process 

•  sufficient  contextual  data 

•  re-check  assumptions,  interpretations,  and  data 
accuracy 

•  pick  up  on  clues  provided  by  behavior  patterns 

•  suspect  everything 

•  relate  chart  signals  to  process  events  and  activities 

•  check  process  compliance 
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-> 

Measure 

|  Analyze  ^ 

Act 

> 

Control 

Methods  to  Change  the  Process 


Improving  a  stable  process  requires  making  changes  to 
common  cause  entities  and  variables.  Selecting  the  right 
change  involves  examination  of: 

•  process  decomposition  and  evaluation 

•  technology  change 

•  cause  and  effect  relationships 

•  products  and  by-products  from  other  processes 

•  business  strategies  and  management  policies 

These  factors  may  well  be  the  drivers  for  changing  the 
process! 
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Define 


> 


Measure 


Tips  for  Changing  the  Process 


Agree  on  process  performance  issues. 

•  What  needs  improvement,  why,  and  how  much? 

Select  process  performance  variables,  target  means,  and 
variability. 

Determine  required  changes  to  common  cause  entities 
and  variables. 


Select  pilot  process. 

Execute  and  measure  the  changed  process. 

Compare  new  process  performance  data  to  historical 
baseline. 

Make  conclusions  and  recommendations. 
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Common  Data  Risks  and  Pitfalls 

Analysis  misses  the  “big  picture” 

Charts  are  colorful,  but  meaningless 

Data  set  lacks  robustness 

No  baseline  for  comparing  current  performance 

Infrequent  comprehensive  rechecks  of  the  data 

Comparing  or  predicting  process  results  without  ensuring 
stability  of  processes 
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A  Process  Improvement  Toolkit 


Benchmark 

Baseline 

Contract/ Charter 
Kano  Model 

Voice  of  the 
Customer 
Voice  of  the 
Business 
Quality  Function 
Deployment 

Process  Flow 
Map 

Project 

Management 

"Management  by 
Fact" 


Measure 

Analyze 

Defect  Metrics 

7  Basic  Tools 

Data  Collection 

Cause  &  Effect 

Methods 

Diagrams,  Matrix 

Sampling 

Failure  Modes  & 

Techniques 

Effects  Analysis 

Measurement 

Statistical 

Sys.  Evaluation 

1  nference 

Quality  of  Data 

Reliability  Analysis 

Root  Cause 
Analysis 

4  Whats 

5  Whys 

Hypothesis  Test 

ANOVA 

1  m  prove 

Control 

Design  of 

Statistical 

Experiments 

Controls: 

Modeling 

•  Control  Charts 

Tolerancing 

Robust  Design 

•  Time  Series 

methods 

Systems 

Non- Statistical 

Thinking 

Controls: 

Decision  & 

•  Procedural 

Risk  Analysis 

adherence 

•  Performance 
Mgmt 

•  Preventive 

activities 
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Section  II:  From  Data  to  Decisions 

This  concludes  our  introduction  to  understanding  data 
and  getting  the  most  use  out  of  your  analyses. 

In  Section  II:  Data  Analysis  Dynamics  we  will 

•  share  our  experiences 

•  provide  useful  tips  for  how  to  make  measurement  work 

•  thread  together  methods,  tools,  processes 

•  provide  a  path  for  action 
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Analysis  Dynamics  1 
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Analysis  Dynamics  2 
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Analysis  Dynamics  3 
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O  typical 


Identify  the  Goals 


stumbling 

point 


Goals  should  be  continuously  generated. 

Without  data,  goals  are  stated  at  a  conceptual  level. 

By  quantifying  performance 

•  problems  are  characterized 

•  true  customer  specifications  are  understood 

•  quantitative  goals  statements  can  be  made 

Typical  problems 

•  goals  do  not  exist  or  have  not  been  explicitly  stated 

•  goals  at  different  levels  are  disconnected 
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Identify  the  Goals  2 

Relevant  tools  and  methods 

•  Voice  of  the  Client 

•  Quality  Function  Deployment 

•  Management  by  Fact 

•  4  Whats 

•  SMART  goals 

•  FAST  diagrams  (Function  Analysis  Systems 
Technique) 
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Identify  the  Goals:  Example 


->  •  scatter  plots  < - - - >•  scatter  plots 

. . . . 
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What  if  there  are  no  “Business 
Goals”? 

Without  high-level  business  goals,  data-driven 
improvement  efforts  quickly  become  fragmented. 

Articulate  business  goals  by 

•  Brainstorming  with  leadership 

•  Organizing  results  into  strategic,  operational  goals 

-  add  in  any  tactics  that  emerged  during  brainstorming 

•  Performing  hierarchical  structure  check 

-  “How?”  answered  top  to  bottom 

-  “Why?”  answered  bottom  to  top 

Verify  that  tactics  drive  impact  and  success. 
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Black  Box  Process  View 

What  are  the  key  inputs  and  outputs  to  your  process? 
What  are  key  in-process  variables  over  which  you  have 
control? 

Typical  problems 

•  Omitting  this  step  -  avoids  examination  of  your 
assumptions  and  understanding  of  the  process 

•  Selecting  a  view  that  matches  the  issue  or  study  level 

•  Constructing  a  view  that  does  not  match  reality 

Relevant  tools  &  methods 

•  Process  Mapping 

•  Mental  Model 
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What  is  a  Process? 


•  Any  set  of  conditions  or  causes  that  work  together  to 
produce  a  given  result 

•  A  system  of  causes  which  includes  people,  materials, 
energy,  equipment,  and  procedures  necessary  to  produce 
a  product  or  service 


Requirements 

Ideas 

Time 


Ml! 


Ijuw*to  Ll/ — 

People  Material  Energy  Equipment  Procedures 

* 


Work  activities 


n 


Products  & 
Services 
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Problem  Management  Process 
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Development  Process  Map 


Design 


O  Requirements 
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O  Concept  design 


> 


Code 

5 

-> 

Compile 

^Resources 

*  Code  Stds 

*  LOC  counter 

f  Interruptions  > 

1 

O  Code 

Unit 

Test 


► 


O  Executable  Code 

•  Test  Plan,  Technique 

•  Operational  Profiles 


•  Detailed  Design 

•  Test  cases 

•  Complexity 

•  Data:  Design  Review 
defects,  Fix  time, 
Phase  duration 


•  Code 

•  Data:  Defects, 
Fix  time,  Defect 
Injection  Phase, 
Phase  duration 


•  Executable 
Code 

•  Data:  Defects, 
Fix  time,  Defect 
Injection  Phase, 
Phase  duration 


•  Functional 
Code 

•  Data:  Defects, 
Fix  time,  Defect 
Injection  Phase, 
Phase  duration 
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A 

Inspection 

O 

Critical  Inputs 

> 

*  Standard  Procedure 

O 
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Rework 

t 

Noise 

•  Control  Knobs 
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O  typical 


Is  the  Data  Right  ? 


stumbling 

point 


Understand  the  data  source  and  the  reliability  of  the 
process  that  created  it. 

Typical  problems 

•  wrong  data 

•  missing  data 

•  accuracy 

•  veracity 

•  credibility 

•  skewed 

Data  transformations 

•  ratios  of  bad  data  still  equal  bad  data 

•  increasing  the  number  of  decimal  places  does  not 
improve  the  data 
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Is  the  data  right?  -  Example 


Inspection  Data  Set  1 


#  of 
People 

Prepare 

tion 

Effort 

Size 

(SLOG) 

5 

3.  7 

2070 

6 

21  .O 

555 

6 

5.1 

1  02 

8 

1  8.0 

260 

6 

1  2.0 

1  Ol 

8 

22.1 

1  65 

6 

1  1  .8 

1  764 

8 

9.2 

348 

5 

7.3 

76 

8 

1  6.5 

1  575 

5 

1  2.5 

2441 

6 

1  8.3 

1  26 

5 

6.5 

88 

6 

7.1 

383 

8 

1  0.2 

111 

8 

1  1  .5 

1  92 

6 

5.2 

212 

7 

9.3 

401 

7 

8.8 

81  5 

5 

31  .O 

551 

5 

4.9 

429 

8 

1  2.7 

883 

9 

30.3 

1017 

8 

26.4 

2116 
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#  of 
People 

Prepara 

tion 

Effort 

Size 

(SLOG) 

4 

2.0 

350 

3 

1  .5 

210 

3 

2.0 

333 

3 

2.0 

430 

3 

2.0 

400 

1 

2.5 

400 

4 

3.0 

440 

3 

2.5 

450 

3 

3.5 

440 

3 

3.0 

255 

3 

2.8 

470 

4 

2.8 

500 

3 

1  .5 

253 

2 

0.7 

78 

4 

7.0 

900 

3 

3.5 

400 

3 

4.8 

1014 

3 

1  .5 

1  20 

5 

1  5.0 

1  495 

4 

4.0 

200 

4 

4.0 

200 

4 

4.0 

200 

3 

4.5 

200 

4 

4.0 

200 
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O  typical 


Do  I  Have  the  Right  Data?  1 


stumbling 

point 


Analyses  can  get  off  on  the  wrong  track  if  the  data  is 
misunderstood,  or  implicit  assumptions  are  made  about  it. 

Analyst  must  ask  questions: 

•  “Do  I  have  measurements  of  all  the  significant  and 
relevant  factors? 

•  “Does  this  data  represent  what  I  think  it  does?” 

Typical  examples 

•  total  SLOC  count  in  place  of  new/changed  SLOC  count 

•  date  recorded  is  often  not  the  same  as  date  observed 

•  use  of  averages  based  on  unstable  processes  (as  in 
normalization) 


©2003  by  Carnegie  Mellon  University 


Version  1.0 


page  42 


Carnegie  Mellon 

Software  Engineering  Institute 


Do  I  Have  the  Right  Data?  2 

Frequently  the  answers  to  these  questions  can  not  be 
answered  by  a  simple  “eyeball”  test,  then  an  initial 
evaluation  must  be  made  using  various  tools  and 
methods. 

Relevant  tools  &  methods 

•  Process  Mapping 

•  Goal-Driven  Measurement  templates 

•  Operational  definitions 

•  Initial  evaluation/exploration  assessment  using 
statistical  tools 
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O  typical 


stumbling 

point 


Initial  Evaluation  /  Exploration  1 


What  should  the  data  look  like? 

•  first  principles  or  relationships 

•  mental  model  of  process  (refer  to  that  black  box) 

•  what  do  we  expect 

What  does  the  data  look  like? 

•  Magnitude,  range,  and  frequency 

•  look  at  absolute  and  percentages 

•  the  shape  of  the  curve 
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O  typical 


stumbling 

point 


Initial  Evaluation  /  Exploration  2 


Relevant  tools  &  methods 

•  descriptive  statistics 

•  run  charts  or  SPC  charts 

•  time  series 

•  boxplots 

•  correlation  plots  -  first  scan  of  relationships 
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Exploration 

Is  this  the  right  data? 

•  unexpected  high 
inspection  rate 

•  unusually  large 
SLOC  per 
inspection 

•  How  many 
inspectors 
contributed  to  the 
prep-hr  effort? 


Example1 


n  r 

Review 

ID 

Defects 

SLOC 

SLOC/ 

RotfHK 

Re\r 

3REP 

Defect/ 

KSLOC 

Defects 

/hour 

30 

9 

9,800 

/ 933. \ 

10.5 

0.918 

0.857 

32 

5 

16,091 

/  804.6\ 

20 

0.311 

0.250 

34 

45 

73,344 

12,716.4' 

27 

0.613 

1.667 

36 

45 

32,352i 

808.8 

40 

1.390 

1.125 

37 

12 

51,525 

5,725.0 

9 

0.233 

1.333 

41 

13 

98,207 

4,214.9 

23.3 

0.132 

0.558 

43 

19 

16,09' 

707.3 

22.75 

1.180 

0.835 

44 

13 

204,216 

8,168.6 

25 

0.064 

0.520 

45 

14 

80,775 

4,895.5 

16.5 

0.173 

0.848 

47 

2 

72,747 

5,914.4 

12.3 

0.027 

0.163 

48 

14 

10,901 

681.3 

16 

1.284 

0.875 

50 

11 

11,468 

1,146.8 

10 

0.959 

1.100 

52 

31 

16,909 

\  573.2/ 

29.5 

1.833 

1.051 

53 

17 

28,538 

\,902.& 

15 

0.596 

1.133 

57 

22 

18,136 

\824 / 

22 

1.213 

1.000 
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Exploration  Example2 


Little  to  no 
correlation 
between 
SLOC  size 
and 

inspection 

effort 

R  =  0.04549 
R2  =  0.00207 


40- 


35- 


30- 


o  25- 


CD  „„ 
■5  20- 

CD 


DC 


15- 


10- 


Scatterplot  of  Review  hours  vs  SLOC  Reviewed 


Y=1 9.06652+1. 28868E-5X 


B 

DATA1.B.LR 


- 1 - 

20000 


40000  60000 

SLOC 


— 1 — 

80000 


- 1 - 

100000 
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Evaluation  Example3 

Given  there  is  no  correlation  between  review  time  and  the 
amount  of  SLOC  reviewed, 

What  questions  can  be  raised  about  the 

•  SLOC  count? 

•  review  time? 

•  number  of  defect? 

•  defect  density? 

•  defects  discovered  per  review  hour? 
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O  typical 


Can  I  Move  Forward? 


stumbling 

point 


Does  the  initial  evaluation/exploration  of  data  support  the 
critical  assumptions? 

What  are  your  assumptions? 

•  are  they  explicitly  articulated? 

•  for  process,  for  data? 

What  are  the  risks  you  are  taking  if  you  move  forward  with 
the  assumptions  you  have  made? 

Is  the  variability  or  presence  of  process  issues  so 
significant  that  they  overshadow  data  issues? 
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O  typical 


Moving  Forward  1 


stumbling 

point 


Moving  forward  is  often  a  judgment  call 

•  can  proceed  with  further  data  and  process  analysis  in 
parallel  with  improving  data 

-  it’s  a  tradeoff  and  a  matter  of  balancing  risks 

•  else  get  the  “right”  data  before  proceeding 

Types  of  actions 

•  removing  assignable  causes 

•  reducing  common  cause  variation 

•  testing  hypotheses 

•  further  decomposing  data  and  process 
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Moving  Forward  2 

This  is  the  “heart”  of  the  analysis 

•  Explore,  establish/confirm  cause-effect  relationships 

•  Plot  trends  over  time 

•  Look  for  and  identify  the  “drivers”  or  dominant  factors 

•  Gauge  the  variation  of  the  variables 

•  Find  assignable  causes 

•  Determine  stability  and  capability  of  processes 

•  Decompose  to  find  root  cause 

Relevant  tools  &  methods 

•  The  “Basic  Tools” 
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Moving  Forward-  Basic  Tools 

Fundamental  data  plotting  and  diagramming  tools 

•  Cause  &  Effect  Diagram 

•  Histogram 

•  Scatter  Plot 

•  Run  Chart 

•  Box  and  Whisker  Plots 

•  Pareto  Chart 

•  Control  chart 

The  list  varies  with  source.  Alternatives  include 

•  Bar  charts 

•  Flow  Charts 

•  Descriptive  Statistics  (mean,  median  and  so  on) 

•  Check  Sheets 
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Moving  Forward- 
Establish  Relationships 


Review  Hours  vs 


Review  Hours  vs 


Total  SLOC 


R  =  0.83569 
R2  =  0.69838 


R  =  0.19244 
R2  =  0.03703 
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Identify  Dominant  Factors 

Profile  of  Defects  Found  in  Product  XYZ 


Syntax  Error  Ambiguous  Interface  Error  Missing  Function  Memory 
Requirements 

Defect  Type 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


page  54 


Cumulative  Percentage 


Number  of  Days 
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Moving  Forward- 
Determine  Extent  of  Variability  1 

Basic  Histogram  shows  Look  for  multimodal  distributions, 

distribution,  spread.  They  point  to  multiple  processes. 


Product-Service  staff  Hou  Time  t0  f'x  a  defect  found  after  development 
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Moving  Forward- 

Determine  Extent  of  Variability  2 


Add  control 
limits  to  reflect 

-t— < 

process  J 

capability  | 

O " 
CD 


Voice  of  the  Process 


LCL=  36.08 


CL=  45.06 


UCL=54.04 


Product  Service  Staff-Hours 
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Add 


Moving  Forward- 

Determine  Extent  of  Variability  3 

k=i  Voice  of  the  Process f  :> 


LCL=  36.08 


CL=  45.06 


UCL=54.04 


Carnegie  Mellon 

Software  Engineering  Institute 


Moving  Forward- 
Find  Assignable  Causes 


Problem  Repair- 

Wait  time 

•  Issue:  Delays  in 
repairing  software 
test  sets 

•  Control  chart 
indicates  process 
unpredictable 

•  Pattern  suggests 
mixture  of  cause 
systems 


2nd  Qtr  and  3rd  Qtr  Problem  Closures 
Wait  time-days 


UCL=292 


20  40  60  80  100 

Problem  Closure  Sequence 
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Moving  Forward- 
Finding  Assignable  Causes 


Problem  Repair-  Wait  time 

Histogram  indicates  data 
includes  possible  mixture  of 
cause  systems 
•One  process  for  problems 
up  to  150  wait  days 
•A  second  process  involving 
more  than  1 50  wait  days 


2nd  Qtr  and  3rd  Qtr  Problem 
Closures  Wait  time 


1 00  200 


300  400 


Wait  time-Days 
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Moving  Forward- 
Find  Assignable  Causes 

Problem  repair  Wait  time  <150  days 


UCL=161.3 

One  process  with  67- 
day  average  wait  time 
•  Near  stable 
•Investigate  cause 
cl=67.29  system  for  driving 

factors 

§  nature  of  defect 
lcl=°  §  staffing 

§  equipment 
§  test  set  type 

Problem  Closure  Sequence 
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Moving  Forward- 
Find  Assignable  Causes 


Problem  Repair  Wait  time  >150  days 


0  5  10  15  20 

Problem  Closure  Sequence. 


UCL=405.2 


CL=246 


LCL=86.82 


Another  process 
with  average  wait 
time  of  246  days 
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Moving  Forward- 
Find  Assignable  Causes 

Problem  Repair-Wait  time 

•  Determined  that  there  were  two  processes  in  operation 

•  Since  both  were  (near)  stable,  necessary  to  examine 
cause  systems  for  components  that  may  be  the  driving 
contributors  to  wide  variation  and  make  appropriate 
changes  to  each  process 

•  Activities  undertaken: 

-  Classification  of  problems  (defects)  reported  and  found 

-  Classification  of  test  sets 

-  Evaluation  of  test  equipment  availability 

-  Availability  of  necessary  skills 
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Decomposition 

Decomposition  is  separating  the  process  into  its 
component  parts  or  data  by  one  or  more  of  its  attributes 

•  Makes  sources  of  variation  visible 

•  Provides  opportunity  for  process  improvement 
This  approach  is  useful 

•  when  process  is  stable  and  process  change  is  needed 
to  reduce  variation 

•  for  highlighting  unusual  data  attributes  that  may  be  the 
source  of  variation 
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Decompose  Data 


•  Defect  data 
decomposed  by 
year 

•May  also 
decompose  by 
project  type, 
organizational 
slices,  and  so  on 


•Means  comparison  test  determines  if  data  groupings  -1 
are  statistically  different.  These  groups  are  not  different. 


•Values  and  sample  size  are  accounted  for  in  the  test. 
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Decompose  Process  Data 

Twenty  one  components  from  same  product,  same  team 

•  approximately  same  size 

•  approximately  same  complexity 

Defects  found  in  design  inspection  are: 


Component 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

Totals 

Defects 

12 

16 

18 

32 

22 

16 

23 

35 

15 

27 

16  25 

20 

26 

20 

23 

23 

36 

22 

27 

17 

471 

Defect  Type 

Number  of  Defects  per  Type  per  Component 

Function 

3 

5 

4 

4 

4 

3 

3 

20 

4 

11 

2 

3 

3 

5 

3 

7 

4 

5 

5 

15 

2 

115 

Interface 

2 

2 

4 

4 

3 

4 

2 

3 

3 

4 

2 

3 

5 

3 

3 

3 

2 

16 

6 

2 

4 

80 

Timing 

1 

1 

0 

1 

1 

0 

2 

1 

0 

0 

2 

0 

1 

1 

1 

1 

1 

0 

1 

0 

0 

15 

Algorithm 

0 

0 

1 

14 

2 

0 

0 

0 

0 

0 

0 

1 

5 

2 

7 

6 

5 

1 

2 

0 

1 

47 

Checking 

1 

1 

5 

1 

7 

1 

1 

2 

0 

1 

6 

3 

1 

12 

1 

0 

2 

4 

3 

5 

2 

59 

Assignment 

0 

2 

0 

4 

1 

2 

1 

3 

2 

3 

2 

8 

1 

0 

2 

1 

2 

1 

0 

1 

1 

37 

Build/Pkg. 

3 

1 

1 

2 

1 

0 

0 

4 

3 

6 

1 

0 

2 

1 

1 

1 

3 

2 

2 

2 

1 

37 

Document 

2 

4 

3 

2 

3 

6 

14 

2 

3 

2 

1 

7 

2 

2 

2 

4 

4 

7 

3 

2 

6 

81 
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Decompose  Process  Data  2 


Apparent  stable  process 
behavior 

•But,  defect  rate  too  high 
and  too  much  variation 

•Explore  examination  of 
defects  by  type 


UCL  =  44.9 


CL  =  22.4 


LCL  =  -0.0448 


UCL  =  27.6 


CL  =  8.45 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


page  66 


Carnegie  Mellon 

Software  Engineering  Institute 


Decompose 
Process  Data  3 

Establish  process  stability 
by  defect  type 

X’s  mark  assignable 
causes  by  defect  type 

Elimination  of  assignable 
causes  will  reduce  variation 
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Potential  Process  Improvement 


Before  Improvement  After  Improvement 


Control  chart  on  right  reflects  potential  improvement 
if  all  assignable  causes  removed 
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Repeat  until . 

Root  cause(s)  found 

The  process  is  at  target,  with  desired  variability 

Other  process  performance  data  has  not  suffered 

•  l.e.  the  process  has  not  been  suboptimized 

Relevant  tools  &  methods 

•  Management  by  Fact 

•  5  Whys 

•  Dashboard 
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Number-Crunching  Tools 


Analysis  done  in.... 

Comment 

Spreadsheet  (Excel) 

•  Most  people  have  a  copy 

•  OK  for  some  basic  charts 

•  Nice  for  presentations 

•  Otherwise  quite  limited 

Excel  Addin 

•  Many  new  add-ins  available 

•  Enables  a  wider  variety  of  charts 

Standalone  SPC 
Package 

•  May  be  better  suited  for  charts  which  an 
organization  is  routinely  monitoring  than  for 
exploration 

Statistical  Package 

•  Higher  learning  curve  than  others 

•  Best  for  those  doing  data-driven  improvement 
as  large  part  of  their  workload 
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Section  III:  Case  Study 

This  concludes  our  introduction  to  analysis  dynamics. 

In  Section  III  we  will  showcase  these  dynamics  through  a 
case  study. 

Context: 

•  organization  project  portfolio  includes  both  new 
development  and  maintenance 

•  project  size  and  complexity  varies  significantly 

•  project  schedules  vary  from  <1  month  to  >18  months 
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Case  Study  Overview 

This  case  study  features  the  following: 

•  pursuit  of  customer  satisfaction 

-  via  proxies  of  defects  and  effort  &  schedule  variance 

•  initial  data  evaluation  and  exploration 

•  initial  data  and  process  decomposition 

•  separation  of  goals  into  “monitor”  and  “improvement” 

•  first  iteration  of  root  cause  analysis  for  improvement 
goals 


Along  the  way,  we  will  use  this  stop  sign 

•  to  pause  and  generalize, 

•  to  ask  probing  questions, 

•  to  extend  the  topic 
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Analysis  Dynamics  1 


Sound  bytes 

“We  didn’t  stumble  here-there  were 
goals  from  the  beginning-but  it  took 
time  to  clarify  them,  to  make  them 
quantitative,  and  to  separate 
monitoring  f ro m  impro vement.” 

“We  had  a  lot  of  missing  data.  We 
conducted  “data  archaeology”  as 
much  as  possible  to  backfill  the  data 
set.  Learnings  were  used  to  improve 
the  automation  of  data  collection.” 


“Our  data  wasn’t  perfect,  but  no 
matter  how  we  sliced  it,  there  were 
clear  improvements  to  pursue.” 
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Analysis  Dynamics  2 


Initial  Evaluation _ 

•  What  should  the  data 
look  like? 


•  What  does  the  data 
look  like? 


Can  I  characterize  the 
process  and  problem? 


Decision  point: 

•  Can  I  address  my  goals 
right  now? 

•  Or  is  additional  analysis 
necessary?  at  the  same  or 
deeper  level  of  detail? 

•  Can  I  move  forward 


Sound  bytes 

•  “We  were  able  to  identify  many  of 
the  “data  rightness”  issues  without 
exploring  the  data.  But,  in  some 
cases,  it  was  necessary  to  dive  into 
the  data  to  identify  the  issues.” 


“For  earned  value  data,  we  found 
the  process  to  be  consistently  “out  of 
spec,”  yet  the  external  customers 
seemed  satisfied.  Reconciling  the 
‘voices’  of  the  process,  external 
customers  and  internal  management 
is  part  of  the  process.” 
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Analysis  Dynamics  3 


Sound  bytes 


“Initial  iterations  of  decomposition  will 
•be  shown.  Because  of  risks 
associated  with  imperfect  data,  each 
conclusion  needs  to  be  carefully 
weighed  against  the  need  for 
additional  verifying  data.” 
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Customer  Data 

•  What  data  are  readily  available  data? 

-post-project  surveys 

•  Data  archeology 

-What  has  been  communicated  via  emails,  phone 
calls 

•  Is  the  data  “perfect”?  NO 

-few  responses 
-qualitative  responses 

•  New  data  collection  needed: 

-updated,  routine  customer  survey 


W 

^ _ f 


By  the  way,  is  data 
ever  perfect?  Can 
you  afford  to  wait  for 
perfect  data? 
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Customer  Data  -  Sample 

Qualitative  comments,  all  positive: 

•  Pleasure  to  work  with! 

•  Outstanding  in  all  aspects!! 

•  If  this  team  had  been  on  this  project  from  the  start  a  lot  of 
things  may  have  gone  smoother. 

•  Really  good  to  work  with.  Have  been  working  with  them  2- 
3  years  now.  They  do  a  good  job  and  we  get  along  well. 

Quantitative  comments 

•  Finished  testing  without  having  to  create  any  additional 
builds. 

•  We  were  able  to  save  three  flights. 
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Defect  Data 

•  What  data  are  readily  available  data? 

-peer  review  inspection  data 

•  Data  archeology 

-field  defects  and  confirmation  of  in-process  defects 

•  Is  the  data  “perfect”?  NO 

-missing  data 

-defect  data  skewed  toward  low  priority  defects 
-variations  in  operational  definitions 
-feedback  loops  at  group  level,  not  org  level 

•  New  data  collection  needed: 

-confirmed  operational  definitions 
-improved  automation  of  data  collection  process 
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Field  Defect  Data  Baseline 


Organizational  goal:  0  field  defects 


Field  defects 


Field  Defects 

FY  01 

4 

FY  02 

4 

In-process  defect  detection 
•  #  of  defects  vs.  development  life  cycle 


When  your  “count  for 
the  year”  is  4,  how 
useful  are  control 
charts? 

And,  if  your  counts 
are  higher? 

Leading  in-process 
indicators  are  what 
you  should  consider 
for  control  charting. 
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Earned  Value  Data 

Readily  available  data 

•  monthly  process  effort,  cost,  schedule 

•  compared  to  specification 

-  with  text  entry  for  out  of  spec  causes 

Data  “archaeology”: 

•  completed  project  data 

-  final  vs.  original  with  differences  categorized 

Is  the  data  “perfect”?  NO 

•  losing  track  of  replanning  impact  on  performance 

•  monthly  data  uses  non-homogeneous  sample 

•  sparse  data  -  some  parts  of  organization  better  represented 

•  not  sure  if  “extreme”  outliers  can  be  excluded 
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Completed  Project  Data  Baseline 


%  effort  variance 

%  sched  variance- 

average 

C-66.1°/0 

<^!5.05S> 

standard  deviation 

'58:3% 

median 

(^0.9%^ 

C-8.1%? 

min  to  max 

-2689.9%  to  50.1% 

-99.8%  to  128.0% 

n 

42 

42 

capability  notes 

(45 £%> 

(spec  =  +/-  20%) 

outside  spec 

oulsTTtS'spec 

This  represents  (initial  plan  -  final  actual) 

•  negative  numbers  are  overruns 

•  schedule  is  in  terms  of  calendar  days 

It  is  the  total  cumulative  variance 


5 

0 

-5 

-10 

-15 

-20 

-25- 

-30 


1.5 


:i 


0.5- 


-0.5- 


■Jatjgl 


3? 


TairiSU 


[R 


customer-requested/approved  changes  are  included 
one  way  or  another,  this  is  what  the  customer  sees 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


page  82 


Carnegie  Mellon 

Software  Engineering  Institute 


Completed  Project  Data  -  Decomposed 


Contribution  to  total  variance,  by  internal/external  categories 


internal/external 

categories 

median 

contribution  toward 
total  effort  (cost) 

#  of 

projects 

reporting 

median 

contribution  toward 
total  schedule 

wprianr^P 

#  of 

projects 

reporting 

internal  project 

<  7  ) 

C-34.32%^ 

•:  4  ) 

internal  organization, 
outside  project 

-1.25% 

5 

C-73.77%J 

V  3  J 

external,  reqt 

-22.48% 

10 

TO 

external,  sched 

0.00% 

15 

-98.36% 

17 

“Internal”  and  “external”  taxonomy  selected  based  on 
“sphere  of  influence  and  control” 

Risk:  while  “internal  causes”  seem  to  be  a  significant 
opportunity,  a  small  number  of  projects  reported  such  causes 
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Explore,  Evaluate  (Plot,  Plot,  Plot) , 


%  effort  variance 

%  sched  variance 

avg 

C  -2%} 

C  13°^ 

std  dev 

ZXZL 

median 

C  2°/<J 

<L  7%^> 

min  to  max 

-95%  to  50% 

-128%  to  71% 

capability  notes 

(^43.8%) 

Q  39%) 

(spec  =  +/-  20%) 

ou!^WFspec 

outsiacTspec 

When  flyers  are  removed 
•Averages  closer  to  target,  spread  narrowed 
•Medians  minimally  affected 
•Still  nearly  as  many  outside  specs 
•Small  “second  peak”  more  visible 


•  What  are  guidelines 
for  removing  flyers? 


Average  vs.  median 


i - 1 

. 

I  Tarqet 

r 

i  LSL 

._l 

1 

©2003  by  Carnegie  Mellon  University 


Version  1.0 


page  84 


Carnegie  Mellon 

Software  Engineering  Institute 


Explore,  Evaluate  (Plot,  Plot,  Plot)  2 


Schedule  Variance  Distribution  to  Time  Series 


same 

data 


Why  not  a 
control  chart? 


Time  series  plot  shows 
•where  in  time  the  contributions  to 
overall  high  variability  occur 
•possible  change  in  variability  over  time 
•where  in  time  the  points  of  the  possible 
“second  population”  occur 
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Explore,  Evaluate  (Plot,  Plot,  Plot)  3 


Schedule  Variance  Time  Series  to  Control  Chart 


Control  charts  also  show 
•possible  second  “population” 

•wide  variability 

But, 

•process  may  just  not  be  stat.  control 

(if  2  populations,  assumption  violated) 

•wide  limits  have  limited  practical  value 

(use  for  off-line  analysis  only  at  this  stage) 

•control  charts  geared  for  monitoring  sustainment  not  improvement 
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In-Process  Cost/Schedule  Data  Baseline 

Organizational  goal  (specification):  +/- 20% 


In  process  effort/cost  data 

•  all  life  cycle  phases,  all  projects,  Oct  -  June  (770+  pts) 


allriaia - - 

extreme  values  excluded 

mean  +/-  3  standard  deviation^ 

-32  +/-  3*423  or^\ 

-1301  to  1237  ^y 

-2  +/-  3*25  or 
-77  to  73 

schedule  capability 

spec  =  +/-  20%  C 

.18%  of  values  outside^spe^ 

1 7%  of  values  outside  spec 

In  process  schedule  data 

•  all  life  cycle  phases,  all  projects,  Oct  -  June  (770+  pts) 


mean  +/-  3  standard  deviations^. 

^7.2654498  +/-  19.23  or^s. 
^64.96  to  50.43  ^y 

capability  notes 

spec  =  +/-  20%  C 

17%  of  values  outside  spec} 
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In-Process  Schedule  Variance  Boxplot 


re 

> 


o 

CO 


1:  oct-01  2:  nov-01  3:  dec-01  4:  jan-02  5:  feb-02  6:  mar-02 

month  label 


Why  a  boxplot 
and  not  an  SPC 
chart? 


Data  reported  monthly  for  all  projects,  cycle  phases 


Conclusion:  need  to  address  variability 


mean- 


/  o  (jci  uoi  mic 

-median:  50th  percentile 


■^25th  percentile 
10th  percentile 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


page  88 


Carnegie  Mellon 

Software  Engineering  Institute 


Are  There  Group  Differences? 


test  for 

significant 

difference 


Schedule  Variance,  all  projects,  Oct  01  to  Jun  02 

Boxes  influenced  by  quantity  of  data,  and  numbers  themselves 

Are  there  statistically  significant  group  to  group  differences:  NO 
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Are  There  Project  Differences? 


40 

30 

20 

10 


ro 

> 


I'ff’!"  “rr^T 

i 

n  "1 . On"  fj® 

:  msrr 

S  1  ^TT 

■ 

o  s  -,11 

I 

Jft  II 

Schedule  Variance,  all  projects  Oct  01  to  Jun  02 


is 

ir 


Each  box  represents  the  timeline  of  an  individual  project 

Are  there  statistically  significant  project  to  project  differences:  YES, 
in  some  cases  (Tukey-Kramer  test  not  shown) 

Conclusion:  Non-homogeneous  sample  (data  from  all  points  along 
“project  timeline”)  was  a  major  contributor  to  the  “significant 
differences”  and  to  the  overall  variability 
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Improving  Sampling  &  Analysis 


Overall  rollup: 

•  group  the  data  by  project  milestones 


Within  project: 

•  identify  different  control  limits  for  each  development 
phase 

•  compare  each  project’s  phase  against  the  history  of 
similar  projects  in  that  same  phase 

•  robust  sample  for  limit  calculations  is  critical 


wider  limits 
for  projects 
in  planning 
phase 


project  cost  index 

1  0 


1  0  J 

Month 


narrower  limits 
for  projects  in 
execution  phase 
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Our  Improvement  Focus 

Two  performance  improvement  priorities,  for  two  different 
portions  of  the  organization 

•  effort  variation  reduction 

•  schedule  variation  reduction 

Additionally,  a  specific  improvement  effort  to  efficiently 
gather  more  complete,  more  consistent  data 

•  needed  to  more  fully  understand  the  magnitude  of 
variability 

•  needed  to  set  exact  (SMART)  improvement  goals 

(Specific,  Measurable,  Agreed  upon,  Realistic,  Timely) 
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Can  We  Address  the  Goals? 

This  is  a  decision  point  in  the  analysis  dynamics. 

Do  we  have  enough  understanding  of  our  data  and 
process?  NO 

Key  questions  at  this  stage 

•  What  are  the  root  sources  of  the  variability? 

•  How  does  the  in-process  variability  provide  an  early 
view  of  the  end-of-project  result? 
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Data  and  Process  Decomposition 

Brainstormed  root  causes  of  variance 

Decomposed  process  into  4  main  subprocesses 

•  mapped  cause  codes  to  process 

•  identified  cause  codes  that  are  resolved  in-process 

Data  archaeology 

•  evaluated  cause  codes  using  historical  data 


risks  of  data 
archaeology  vs. 
starting  anew 
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Cause  Codes 

Transformed  original  brainstorm  list 

•  initial  experiential  assessment  of  frequency,  impact  of  each 
cause  code 

•  refined  “operational  definitions”  and  regrouped  brainstorm  list 

•  tagged  causes  to  historical  data 

•  refined  again 

Final  list  included  such  things  as 

•  Missed  requirements 

•  Underestimated  task 

•  Over  commitment  of  personnel 

•  Skills  mismatch 

•  Tools  unavailable 

•  EV  Method  problem 

•  Planned  work  not  performed 

•  External 

•  Other 


Direct  Cause  vs. 
Root  Cause 


Causes  resolved  in- 
process  vs.  causes 
that  affect  final 
performance 
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Four  High-Level  Processes  that 
Influence  Final  Performance 


Project  Management 

-Workload  Proposal 
-Planning 

-Requirements  Management 
-Configuration  Management 
-Decision  Analysis  and  Resolution 
-Training 

Organizational  Management 

-Workload  Agreements 
-Resource  Allocation 
-Funding 
-Training 

Project  Monitoring  and  Control 

-Measurement 
-Quality  Assurance 
-Peer  Review 

Technical  Processes 

-Design 
-Implement 
-Formal  Test 
-Release 

Cause  Codes  were  mapped  to  these  processes 
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Prioritizing  the  Causes 


•  Pie  Chart  vs.  Pareto? 

•  Does  everyone  understand 
where  the  data  came  from? 

•  Are  the  algorithms  and 
assumptions  valid? 

•  What  are  the  risks? 


Algorithms  and  Assumptions 

•  frequency  &  impact  of 
occurrences  -  and  which 
occurrences? 

Cause  Codes 

•  Which  are  resolved  in 
process? 

Sphere  of  Influence 

•  internal  vs.  external 

•  degree  of  “process 
understanding” 

•  degree  of  “process  control” 
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Data  Treatments 


Project 

Month 

Cause  Code 

Variance 

Repeat? 

A 

1 

4 

4 

A 

2 

4 

3 

Y 

A 

3 

5 

7 

B 

1 

2 

2 

B 

2 

B 

3 

4 

4 

C 

1 

5 

8 

C 

2 

C 

3 

Cause  Code 

frequency 

impact 

(average) 

frequency  x 
impact  (or  sum) 

2 

1 

2 

2 

4 

2 

4 

8 

5 

2 

7.5 

15 

Cause  Code  data  may 
be  summarized  by 
frequency  (f),  impact  (i), 
or  f  x  i. 

Usage  of  the  latter 
resembles  methods 
used  to  evaluate, 
mitigate  risk — 


▼ 


Risk  Mitigation  Analogy 

frequency 

impact 

H 

H 

M 

M 

L 

L 
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Co-Optimizing  Across  the 
Organization  -  Internal  Causes 


Impact 
#  (from 
Pareto) 

Schedule 

Effort 

Organization 
Slice  1 
Schedule 

Organization 
Slice  1  Effort 

Organization 
Slice  2 
Schedule 

Organization 
Slice  2  Effort 

1 

Under 

estimated 

Task 

Tools 

Under 

estimated 

Task 

Under 

estimated 

Task 

Tools 

Tools 

2 

Tools 

Assets  not 
available 

EV  Problems 

Under 

planned 

rework 

Skills 

mismatch 

Under 

estimated 

Task 

3 

EV  Problems 

Under 

planned 

rework 

Missed 

requirements 

Missed 

requirements 

Under 

estimated 

Task 

Missed 

requirements 

4 

Missed 

requirements 

Planned 
work  not 
performed 

Under 

planned 

rework 

EV  Problems 

Missed 

Requirements 

Unexpected 
departure  of 
personnel 

5 

Skills 

Mismatch 

Under 

estimated 

task 

Asset 

availability 

Planned  work 
not 

performed 

Unexpected 

departure 

EV  Problems 
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In-process  data  as  leading  indicator 


internal/external 

categories 

median 

contribution  toward 
total  effort  (cost) 
variance 

#of 

projects 

reporting 

median 

contribution  toward 
total  schedule 
variance 

#of 

projects 

reporting 

In-f 

freq 

process  c 

impact 

Jata 

f  x  i 

internal  project 

-30.83% 

7 

-34.32% 

4 

internal  organization, 
outside  project 

-1.25% 

5 

-73.77% 

3 

external,  reqt 

-22.48% 

10 

-20.20% 

10 

external,  sched 

0.00% 

15 

-98.36% 

17 

Join  the  views  of  completed  project  performance 
and  in-process  performance. 


Since  “cause  categories”  differ  between  the  data 
sets,  the  first  iteration  is  not  trivial 
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SMART  Schedule  Variance  Goal 

Reduce  the  total  variance  by  decreasing  the  variance  of 
the  top  3  internal  causes  by  50%  in  1  year 

Reduce  the  impact  of  external  causes  by  50% 

Indicators: 

•  Trend  for  each  cause  independently 

•  Trend  for  total  variance 


Will  focus  on  these  causes 
give  us  bottom  line  results? 
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Schedule  Variance  Root  Cause 


Cause  Code: 

Underestimated  tasks 

Process: 

Project  Management 

Subprocesses: 

Planning 

•  Establish  requirements 

•  Define  project  process 

•  Perform  detailed  planning 

Requirements  Management 

As  subprocesses  are  explored,  process  mapping  techniques 
may  be  used  with  (or  based  on)  ETVX  diagrams 
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Schedule  Variance  Root  Cause  2 

Root  Causes  of  Common  Cause  Variation 

•  Inexperience  in  Estimation  process 

•  Flawed  resource  allocation. 

•  Inexperience  in  product  (system)  for 
estimator 

•  Requirements  not  understood 

Root  causes  of  Special  Cause  Variation 

•  Too  much  multitasking 

•  Budget  issues 

A  list  of  possible  countermeasures  was 
developed 


Pros/Cons  of  doing 
this  retrospectively 
vs.  real  time 


What  is  needed 
before  executing  the 
countermeasures? 

Could  the  “special 
causes”  also  be 
“common  causes”? 
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Putting  it  all  Together 

Dashboard  to  monitor  “the  whole  picture” 

•  customer  satisfaction 

•  defects 

•  effort  and  schedule  variance 

Management  by  Fact*  to  monitor  improvement  efforts 

•  effort  variance  reduction 

•  schedule  variance  reduction 

•  measurement  quality  improvement 

Reference  process  documentation 
and  project  management  principles 
in  use. 


Who  uses  the 
dashboards 
and  MBFs? 


*T ooltip  for  Management  by  Fact  (MBF)  in  the  Addendum 
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Notional  Management  by  Fact  (MBF) 


Reduce  the  total  schedule  variance  by  decreasing  the 
variance  of  the  top  3  internal  causes  by  50%  in  1  year. 


Total  variance  w/ 
mean  comparison 


Variance  for  top  3  causes: 

•  Underestimated  Tasks 

•  EV  Method  Problem 

•  Missed  Requirements 


Prioritization  & 

Counter  Measures 

Impact,  Capability 

Root  Cause 

First:  Gather  realtime  data  and 

In  total,  these 

•  Inexperience 

verify  “data  archaeology” 

countermeasures  will 

•  Resource  Allocation 

Then: 

remove  1 5%  of  typical 

•  Requirements  not 

• 

variance. 

understood 

• 

(as  possible,  list  impact  of 

• 

each  countermeasure) 

Still  needed:  Relate  in  process  and  completed  project  data 
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Notional  Dashboard 


Earned  Value  Data 

In-process  data: 

•  monthly  schedule  index, 
cost  index  by  project 
milestones 


Completed  projects  data: 

•  control  chart 

•  %  outside  spec 

•  contribution  of  internal 
causes  to  completed  project 
variance 


Defect  Data:  Tally  of  Field  Defects 

Note:  In-process  profiles  to  be  shown  on  “group  level” 
dashboards 


Customer 

Satisfaction 

F 

1 

Return  on 
nvestment 

ROI 

“return”  =  variance 
reduction  translated  into  $$ 

Measurement 

Quality 


Other  possible  inclusions: 

•  Engineering  process  procedural  adherence  (as  a  leading 
indicator  for  EV,  defect  and  measurement  quality  performance) 
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Organization  Specific  Process 


Select 

Business  Goal 
(Customer 
Satisfaction) 


Business  Objective 
Specs 

Performance  Thresholds 

- ► 


Establish  capability,  models,  etc. 


No  “Issues” 


Identified 

Thresholds 


Gather 

Data 


Analyze  Data  -pi- 


•Project  Performance 
•Measures  Quality 
•SPI  Implementation 


Prioritize 
Issues 

Start  subprocess 
selection 


Improvements 


•Snapshot  (1st  Iteration  Baseline) 
•Issues  (Validity  of  data,  Quality  of 
Data,  Variance  (performance) 


Draft  Improvement 
Goal  (SMART)  or 
Identify  focus  area 


Identify 

Possible 

Causes 

(Brainstorm) 


Gather  Data/Analyze  Data 

Perform  Causal 
Analysis  (OPP) 


Prioritize 

Actual 

Causes 


Identify 

Potential 

Solutions 


1st  Iteration  Final  Goal 


Goal  Refinement 


Develop 

Implement 

Action  Plan 

W 

Improvement 
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Case  Study  Summary  1 


->  • 


>  • 


iterative, 

the 

“dynamics” 

overlap 


Goal:  Customer  satisfaction  via  effort,  schedule,  field  defects 
Black  Box  Process:  not  explicitly  dealt  with  until  root  cause 
Right  Data: 

-  in-process  data  available  sMARTer, 

-  needed  to  “data  mine”  for  completed  data  more 

-  some  “new  data  needs”  identified  quantitative 

Data  is  Right 

-  multiple  iterations  to  correct  some  data  (is  this  in  slides?) 
Explore/Evaluate 

-  key  to  determining  need  for  “data  archaeology” 

-  put  field  defects  into  “monitor”  mode 

-  focus  on  improving  effort,  schedule  variability  (or  change 
specs) 

-  focus  on  improving  measurement  quality 

-  focus  on  improving  sampling  schemes 
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Case  Study  Summary  2 


Explore/Evaluate  continued 

-  extent  of  variability  characterized 

-  some  decomposition  conducted  to  distinguish 
overall  variability  vs.  multiple  populations 

Data  &  Process  Decomposition  * 

-  Sub  processes  of  interest  selected  based  on  pareto 
analysis  of  “cause  codes 

Root  Cause  Analysis: 

-  many  direct  causes  identified 

-  separating  common  and  special  causes  of  variability 

-  we’re  getting  there.... 


decomposition 
starts  in  “initial 
exploration” 
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Case  Study  Summary  -  Tools  Used 


Benchmark 

Baseline 

Contract/ Charter 
Kano  Model 

Voice  of  the 
Customer 

Voice  of  the 
Business 

Quality  Function 
Deployment 

Process  Flow 
Map 

Project 

Management 

"Management 
by  Fact" 


control  charts  for  limited  analysis  NOT  as  control  mech. 


Measure 


Defect 

Metrics 

Data 

Collection 

Methods 

Sampling 

Techniques 

Measurement 
Sys.  Evaluation 

Quality  of 
Data 


adapted 
technique  for 
impact 
evaluation 


Analyze 


7  Basic  Tools 

Cause  &  Effect 

Diagrams, 

Matrix 


Failure  Modes  & 
Effects  Analysis 


Statistical 
I  nference 

Reliability  Analysis 

Root  Cause 
Analysis 

4  Whats 


5  Whys 


Hypothesis  Test 

ANOVA 


I  m  prove 


Design  of 
Experiments 

◄- 


Modeling 

Tolerancing 

Robust  Design 

Systems 
Thinking 

Decision  & 
Risk  Analysis 


anticipate  future 
use  for  these 
improvement 
efforts 


Control 


Statistical 

Controls: 


Control  Charts 


•  Time  Series 
methods 

Non-Statistical 

Controls: 

•  Procedural 
adherence 

•  Performance 
Mgmt 

•  Preventive 
activities 


bold  =  tool  used 
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Summary  -  Key  Points 

Show  me  the  data!  Follow  the  data! 

Couple  data  analysis  with  your  knowledge  of  the  process. 

If  your  number-crunching  is  not  adding  value,  then  STOP! 

•  Plave  a  goal:  a  monitoring  goal,  an  improvement  goal 

This  isn’t  that  hard. 

•  Slow  down,  think  about  your  process  and  proceed 
methodically 

But  it  isn’t  that  easy  either.  (If  it  were,  we’d  all  be  out  of  a  job). 

•  Don’t  be  afraid  to  explore  your  data,  to  pursue  your  ideas. 

Use  your  goals  and  your  data  as  your  guides. 

You  can  get  yourself  into  a  chicken-and-egg  argument  with  data. 

•  Sometimes,  you  need  to  just  dive  in  with  what  you  have. 
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Contact  Information 

Bill  Florae 

Software  Engineering  Institute 

Software  Engineering  Measurement  and  Analysis 

Email:  waf@sei.cmu.edu 

434-978-7780 

Jeannine  Siviy 

Software  Engineering  Institute 
Measurement  &  Analysis  Initiative 
Email:  jmsiviy@sei.cmu.edu 
412-268-7994 

Contact  us  for  a  copy  of  the  slides. 

Or,  leave  a  business  card  with  Jeannine  or  Bill. 
Also,  they  will  be  posted  on  the  SEMA  web  pages 
http://www.sei.cmu.edu/sema 
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Additional  Reading 

References  on  statistics  and  analytical  tools  (URL’s  subject  to  change  without  notice) 

General  Statistics  and  Tools 

Davis,  Wallace  III,  Using  Corrective  Action  to  Make  Matters  Worse,  Quality  Progress,  October 
2000 

Gonick,  Larry  and  Smith,  Woollcott,  The  Cartoon  Guide  to  Statistics,  HarperPerennial,1993 
The  Memory  Jogger,  Goal/QPC,  http://www.goalqpc.com 

Wheeler,  Donald,  J.  Understanding  Variation  -  The  Key  to  Managing  Chaos,  SPC  Press,  1993 
Statistical  Process  Control 

AT&T  /Western  Electric  Co.,  Statistical  Quality  Control  Handbook,  Delmar  Printing  Company 

Chrysler,  Ford,  General  Motors  Corp.,  Statistical  Process  Control  -  SPC,  A.I.A.G.  1995 

Florae,  William  A.,  and  Anita  D.  Carleton,  Measuring  the  Software  Process,  Addison-Wesley, 

1999 

Wheeler,  Donald,  and  David  S.  Chambers,  Understanding  Statistical  Process  Control,  SPC  Press, 
1992 

Wheeler,  Donald  and  Polling,  Sheila,  Building  Continual  Improvement,  SPC  Press,  1998 
Bayesian  Modeling: 

Fenton,  Norman  and  Martin  Neil,  Software  Metrics:  Roadmap,  International  Conference  on 
Software  Engineering,  2000,  available  at  http://www.softwaresvstems.org/future.html 
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Addenda 

Additional  vignettes 
Tool  tips 
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Example  of  an  Aid 
for  Operational 
Definitions  using 
Orthogonal 
Classification 


Reference 
Page  33 


Problem  Status 

Open 

R  ecog  n  ized 

E  va  lu  ated 

R  eso  Ived 

C  losed 

Include 

Exclude 

Value  Count 

A  rra  y  Count 

J 

y 

y 

y 

y 

J 

y 

Problem  Type 

S oftw are  d efect 

Requirements  defect 

Design  defect 

Code  defect 

Operational  document  defect 

T est  case  defect 

Other  work  product  defect 

Other  p rob le m  s 

H  ard  w  a  re  p  ro  b  lem 

Operating  system  problem 

U  ser  m  istake 

O  perations  m  istake 

New  requirement/enhancement 

U  nd  eterm  in  ed 

Not  re  p  eata  ble/C  a  us  e  unknown 
Value  not  id e ntified 

Include 

Exclude 

Value  Count 

Array  Count 

✓ 

y 

✓ 

y 

✓ 

✓ 

✓ 

y 

y 

✓ 

✓ 

✓ 

✓ 

y 

y 

V 

Uniqueness 

O  rig  in  al 

D  uplicate 

Value  not  id e ntified 

Include 

Exclude 

Value  Count 

Array  Count 

J 

y 

y 

y 

Criticality 

1  st  leve  1  (most  critical) 

2nd  level 

3rd  level 

4th  level 

5th  level 

Value  not  id e ntified 

Include 

Exclude 

Value  Count 

Array  Count 

/ 

y 

y 

y 

y 

y 

V 

V 

y 

y 

y 

U  rgency 

1st  (most  urgent) 

2nd 

3rd 

4th 

Value  not  id e ntified 

Include 

Exclude 

Value  Count 

Array  Count 

y 

y 

✓ 

✓ 

y 
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Compliance 

Issues 

May  be  basis  for 
assignable  causes 


Compliance  Issues 

Things  to  Examine  When  Seeking 
Reasons  for  Noncompliance 

Adherence  to  the  process 

awareness  and  understanding  of  the 
process 

existence  of  explicit  standards 

adequate  and  effective  training 

appropriate  and  adequate  tools 

conflicting  or  excessively  aggressive 
goals  or  schedules 

Fitness  and  use  of 
people,  tools,  technology 
and  procedures 

availability  of  qualified  people,  tools, 

,  and  technology 

experience 

education 

training 

assimilation 

Fitness  and  use  of 
support  systems 

availability 

capacity 

responsiveness 

reliability 

Organizational  factors 

lack  of  management  support 

personnel  turnover 

organizational  changes 

relocation 

downsizing 

disruptive  personnel 
morale  problems 
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Initial  Control  Chart  of  Inspection 
Package  Review  Rate  (SLOC/Prep-Hr) 


Inspection  Package  Review  Rate 
All  Product  Components 


Inspection  Sequence  Number 


UCL=676.1 
CL=1 19.2 
LCL=-437.7 


UCL=684.1 

CL=209.4 


Assignable 
causes  due  to: 

•  Erroneous  and 
Missing  Data 

•  Multiple  Cause 
Systems  (six 
components 
each  with  own 
development 
team) 
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Inspection  Package  Review  Rate 
for  Component  A 


Inspection  Package  Review  Rate 


Re-analyzed  data 

•  Data  errors 
eliminated 

•  Consider  single 
major  cause  system 
at  a  time 

•  Control  chart  for 
one  component 

•  Several  assignable 
causes  apparent 
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Component  A  Review  Rate 


Inspection  Package  Review  Rate 
Component  A 


Investigation  resulted 
in  removal  of  separate 
cause  systems  included 
in  inspection  packages: 

•  data  tables 

•  lists 

•  arrays 

•  different  review 
process 
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Component  A  Revision 


Component  A 


Inspection  Sequence  Number. 


Version  1.0 


Process  Instability: 
Apparent  shift  of 
process  performance 
after  #1 5 

Leads  to  investigation 
of  changes  in  process 
cause  systems 
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Cause-and-Effect  Relationships 


400  -i 


Inspection  review  rate  with  increase  in  SLOC 


o- 


n - ■ - 1 - ■ - 1 - ■ - 1 - ■ - 1 — 

0  200  400  600  800 


Y=1 4.6831 5+0.22702  x 


CORRELATION 
LINEAR  0.77 

EXPONENTIAL  (2)  0.86 
EXPONENTIAL  (3)  0.86 


1000 


ModSloc 
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Component  A  Review  Rate 

Distribution  of  review  rates  by  SLOC  size 


x 

£Z 

O 

O 

CD 

CL 

C/5 

C 

O 

o 

_l 

co 

05 

ro 

X 

5 

CD 

> 

CD 

X 
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o 

o 

CD 

CL 

C/5 

C 


Amount  of  SLOC  in  review  package  is  key  driver  of  review  time  spent 

100 

90 
80 
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Inspection  Package  Size  (New  and  Changed  SLOC) 
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Component  A  Review  Rate 


Component  A:  Package  Size  <  60  SLOC 


Inspection  Sequence  Number 


Replot  data  using  two 
charts: 

•  Rates  for  Inspection 
<60  SLOC 

•  Rates  for  Inspection 
>60  SLOC 


Component  A:  Package  Size  >  60  SLOC 


Indicates  two  processes 
in  operation  depending 
on  size  of  Inspection 
package 

Establish  Trial  Limits  for 
each  subprocess 
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Component  A  Review  Rate 


Component  A:  Package  Size  <  60  SLOC 


UCL=29.48 


CL=10.71 
LCL  =  0 


Component  A:  Package  Size  >  60  SLOC 


Additional  observations 
identify  more  assignable 
causes 

Investigation  determines 
that  assignable  cause 
observations  from 
re-inspection  process 


Version  1.0 


page  125 


©  2003  by  Carnegie  Mellon  University 


Carnegie  Mellon 

Software  Engineering  Institute 

Component  A  Review  Rate 


Component  A:  Package  Size  <  60  SLOC 


UCL=29.48 


CL=10.71 
LCL  =  0 


All  re-inspection 
data  identified  and 
removed  from 
control  chart  since 
they  represent  a 
different  process 
(different  cause 
system) 
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Component  A  Review  Rate 


Component  A:  Package  Size  <  60  SLOC 


Charts  plotted 
with  remaining 
data  (single  cause 
system) 


Component  A:  Package  Size  >  60  SLOC 


- , - , - , - , - , - , - , - ,  LCL=  0 

23456789  10 

Inspection  Sequence  Number. 


UCL=154.5 


CL=61 .95 


Additional  data 
points  reinforce 
trial  limits 
hypothesis 
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Component  A  Review  Rate 

Analysis  summary: 

•  Inspection  process  consists  of  several  (3) 
undocumented  subprocesses 

•  Review  rate  appears  to  be  stable  within  two  categories 
(<  and  >  60  SLOC) 

•  Inspection  packages  of  60  SLOC  or  more  reviewed 
about  6X  faster  than  those  with  <60  SLOC 

Key  questions  requiring  more  study: 

•  Why  difference  in  review  rates? 

•  Is  there  a  difference  in  effectiveness  (rate  of  escaped 
defects)? 

•  Do  other  components  behave  similarly? 

•  How  do  rates  compare  from  release  to  release? 
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Tool  Tips  Part  1 :  The  Basic  Tools 

Overview  (description,  procedure,  tips,  examples)  for 

•  run  charts 

•  spc  charts 

•  boxplots 

-  including  pareto  boxes 

•  scatter  plots 

•  histograms,  distributions  and  capability 

-  twist:  rayleigh,  weibull  distributions 

•  bar  charts 

•  pareto  charts 

•  cause&effect  diagram 

-  including  cause  &  effect  matrix 
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Tooltip:  7  Basic  Tools 

Description 

•  Fundamental  data  plotting  and  diagramming  tools 

-  Cause  &  Effect  Diagram 

-  Histogram 

-  Scatter  Plot 

-  Run  Chart 

-  Flow  Chart 

-  Brainstorming 

-  Pareto  Chart 

•  The  list  varies  with  source.  Alternatives  include 

-  Statistical  Process  Control  Charts 

-  Descriptive  Statistics  (mean,  median  and  so  on) 

-  Check  Sheets 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


page  130 


Carnegie  Mellon 

Software  Engineering  Institute 


7  Basic  Tools:  Usage 

Plot  trends  over  time 
Examine  relationships  among  measures 
Explore  cause-effect  relationships 
Prioritize  issues 

Determine  stability  and  capability  of  processes 
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7  Basic  Tools:  Chart  Examples 


90 
80 
70  - 
£*60 
£50 
|  40 
0  30 
20 
10 
0 


Defects  Removed  By  Type 


Pareto  Chart 

✓ 


40  80  50  10 


Defect  Type  Codes 


Run  Chart 


Mean  Time  To  Repair 
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7  Basic  Tools:  Chart  Examples  2 
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7  Basic  Tools:  Cause  &  Effect 


Management 

People 

Unrealistic  \ 

Minimum  \ 

Traditional  diagram 


Process  -Problem 


[Westfall] 
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7  Basic  Tools:  Chart  Variations 


Box  &  Whisker  Plot 
for  assessment  data 


mean 


90th  percentile 
75th  percentile 
median:  50th  percentile 
25th  percentile 
10th  percentile 
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7  Basic  Tools:  Chart  Variations 

Boxplot  variations: 

•  cost  and  schedule  variance  over  time  to  show 
organizational  average  and  also  variability 

•  prioritized  features  for  a  new  process  technology  rollout: 
a  combination  “pareto-boxplot” 


1 

JL 

V 

V 

X  -j 

r  x 

X 

X 

X 

* 

V 

X 

x_ 

X  J 

£ 

X 

X 

X 
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potential  (specific)  process  features 
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Tool  Tip:  Run  Charts 

Description 

•  Time  series  plot  that  can  be  used  to  examine  data 
quickly  and  informally  for  trends  or  other  patterns  that 
occur  over  time. 

Tips 

•  Run  charts  are  not  control  charts  -  don’t  over-interpret 
them. 

•  If  observations  are  not  all  similarly  spaced  in  time, 
there  may  be  more  than  one  process  influencing  what 
appears  to  be  a  single  run. 
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Run  Charts:  Example 


Observation  Number 

Assumptions 

•  ordered  by  time 

•  single  underlying  process 

•  consistent  operational  definitions 
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Tool  Tip:  Statistical  Process 
Control  (SPC)  Charts 

Description 

•  run  chart  with  statistical  limits 
Usage 

•  let  you  know  what  your  processes  can  do,  so  that  you  can 
set  achievable  goals. 

•  provide  the  evidence  of  stability  that  justifies  predicting 
process  performance. 

•  separate  signal  from  noise,  so  that  you  can  recognize  a 
process  change  when  it  occurs. 

•  distinguishes  common  cause  variation  from  special  cause 
variation 

•  point  you  to  fixable  problems  and  to  potential  process 
improvements 
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Control  Chart  Basics 


Specification 
Limits 


CL  +  3c? 


CL  -  3c? 


Upper  Control  Limit 
(UCL) 


Mean  or  Center  Line  (CL) 


-> 


-Lower  Control  Limit 
(LCL) 


Event  Time  or  Sequence 


Limits 


Control  Limits  — >•  Determined  by  Process  Performance  Measurements 

(Voice  of  the  process) 

Specification  Limits  - >-  Set  by  customer,  engineer,  etc. 

(Voice  of  the  customer) 
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SPC:  Tips 

Reacting  to  Common  Cause  Variation  as  if  it  were  Special  Cause 
Variation  cannot  improve  the  process  and  will  result  in  increased 
variability. 

Check  your  data  distributions! 

•  Defect  counts  are  never  negative  and  may  not  be  normally 
distributed. 

Set  specification  limits  based  on  statistics,  engineering 
knowledge  and  risk  of  escaping  defects. 

Implement  charts  “in  the  field”  only  when  you  have  corrective 
action  guidelines.  Otherwise,  work  the  charts  offline. 

Always  look  at  the  average  (or  individual)  arid  range  charts! 
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SPC 


Number  of 
Unresolved 
Problem 
Reports 


Moving 

Range 


Example 


UCL  =  31 .6 

CL  =  20.04 

LCL  -  8.49 

UCL  =  14.2 

CL  =  4.35 
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SPC:  Rules  for  Detecting 

Process  Instabilities  TEST  1 : 


Zone  B 


TEST  3: 

4  out  of  5 
signals  in 
zone  B 


TEST  4: 

8  successive 
points  on 
same  side  of 
centerline 

(£ 

■g 

> 

c 


Single  point  outside  of  zone  C 

UCL 
Zone  C 

B 

Zone  A 

CL 


Zone  A 


Zone  C 
LCL 


Subgroup  No. 


TEST  2: 


2  out  of  three  beyond  zone  B 
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SPC:  Anomalous  Patterns 

„  .  ,  ,  ,  Unstable  Mixture 

Rapid  Shift  in  Level 
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Tooltip:  Scatter  Plots 

Description 

•  Display  empirically  observed  relationships  between 
two  measures. 

Usage 

•  A  pattern  in  the  plotted  points  may  suggest  that  the 
two  measures  are  associated. 

•  When  the  conditions  warrant,  scatter  diagrams  are 
natural  precursors  to  regression  analyses. 

•  Scatter  diagrams  are  rarely  used  as  the  only  means  of 
characterizing  the  relationship  between  two 
measures. 

•  Does  not  predict  cause  and  effect  relationships 
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Scatter  Plot:  Example  with  Line 


Inspection  Effort  versus  Inspected  Sloe 


Y=2.0811 2+0.00435  x 

R  =  0.45031 
R-Squared  =  0.20278 


sloe 


R  =  Correlation  Coefficient 


R-Squared  =  Fraction  of  variability  explained  by  the  model 
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Tooltip:  Histograms 

Description: 

•  Display  the  empirically  observed  distribution  for 
values  of  a  measure. 

•  Show  the  frequency  of  each  value  and  the  range  of 
values  observed. 

Usage: 

•  Inappropriate  unless  the  measure  can  be  treated  as  a 
continuous  scale. 
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Histograms  -  Examples 


Product-Service  Staff  Hours 


Look  for  multimodal 
distributions 
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Tooltip:  Bar  Charts 

Description 


Usage 

•  Similar  in  many  ways  to  histograms 

•  Do  not  require  that  the  measure  be  treated  as  a 
continuous  variable. 

•  Bar  charts  are  much  more  frequently  used  than 
histograms. 
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Bar  Charts:  Example 


Defect  Analysis 


C/5 

O 

ffi 

05 

Q 

c 

05 
O 
■ — 

CD 

CL 


analysis  test  test  test  use 


Software  Activity 
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Bar  Charts:  A  Word  of  Caution 


Because  they  are  so  flexible,  it  is  easy  to  get  carried 
away  with  bar  charts. 


Defect  Counts  by  Project  and  Type 


o 

fl) 

0 

Q 


25 

20 

15 

10 

5 

0 


□  Syntax  Error 

■  Ambiguous  Requirements 

□  Interface  Error 

□  Missing  Function 
M  Memory 


A  B  C  D  E  F 


Project  Identifier 
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Tooltip:  Pareto  Charts 

Description: 

•  Special  form  of  a  bar  chart  that  ranks  categories  of  data  in 
terms  of  their  amounts,  frequency  of  occurrence,  or 
economic  consequences. 

Usage: 

•  Ranking  of  problems,  causes,  or  actions,  etc.,  must  be 
orthogonal 

•  Interpretation  based  on  the  “80/20  rule” 

If  the  80/20  rule  does  not  apply 

•  Consider  counting  a  different  attribute,  while  maintaining  the 
same  stratification. 

•  Consider  re-stratifying  -  use  a  different  classification 
scheme. 

•  Consider  a  different  attribute  of  the  process  under  study. 
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Pareto  Charts:  Example 


Profile  of  Defects  Found  in  Product  XYZ 


Syntax  Error  Ambiguous  Interface  Error  Missing  Function  Memory 
Requirements 


Defect  Type 
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Tooltip:  Cause  &  Effect  Diagrams 


Description 

*Also  called  “Fishbone”  or  “Ishikawa”  diagrams) 

Usage 

•  Allow  you  to  probe  for,  map,  and  prioritize  a  set  of 
factors  that  are  thought  to  affect  a  particular  process, 
problem,  or  outcome. 

•  Help  elicit  and  organize  information  from  people  who 
work  within  a  process  and  know  what  might  be 
causing  it  to  perform  the  way  it  does. 
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Cause  &  Effect  Diagram:  Tips 

You  can  spend  a  lot  of  time  discussing  what  the  principal 
causes  should  be  (the  main  branches)  if  you  are  not 
careful. 

•  May  need  to  work  on  the  categorization  of  causes  in 
advance 

•  May  want  to  just  use  generic  cause  categories  like; 
Materials,  Equipment,  Operators,  Procedures  and 
Environment. 
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Cause  &  Effect  Diagram:  Example 


problem  reports 
not  logged  in  properly 


cannot  isolate  software 
artifact(s)  containing 
ithe  problem 


change  control  board  meets 
once  a  week 


changes  decisions 
not  released  in  a 
timely  manner 


delays  in  approving 
^changes 


It  takes  too 
long  to 
process  our 
software 
change 
requests 


information  missing 
from  problem  reports 


cannot  determine 
what  needs  to  be  done 
to  fix  the  problem 


takes  time  to 

make  changes 


delays  in  shipping 
changes  and  releases 

delays  enroute 


'cannot  replicate 

problem 


must  reconfigure 

baselines 


©  2003  by  Carnegie  Mellon  University  Version  1 .0 


page  156 


Carnegie  Mellon 

Software  Engineering  Institute 


Basic  Tools:  Cause  &  Effect 


Management 

People 

Unrealistic  \ 

Minimum  \ 

Traditional  diagram 


Process  -Problem 


[Westfall] 
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Tool  Tip:  Cause  &  Effect  Matrix 

Description 

•  method  to  determine  possible  causes  of  variation  in  the 
process  and  to  feed  future  experimental  designs 

Purpose 

•  to  organize  problem-solving  efforts  when  there  are 
multiple  responses  involved 

•  to  prioritize  the  number  of  factors  to  study 

•  to  build  team  consensus  about  what  is  to  be  studied 


[Hexsab  02] 
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Cause  &  Effect  Matrix:  Usage 

When  to  use: 

•  team  is  overwhelmed  with  the  number  of  variables  affecting 
process 

•  not  possible  to  experiment  with  all  of  the  variables  -  need  to 
narrow  down  the  list 

•  team  is  struggling  with  which  factors  may  have  the  biggest 
impact 

•  it  is  not  clear  how  each  factor  impacts  customer  requirements 
Feeds  other  tools: 

•  Failure  Mode  and  Effects  Analysis 

•  Data  collection  plans 

•  Experimentation 

•  Control  plans 

[Hexsab  02] 
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Cause  &  Effect  Matrix:  Terms 


Process:  The  combination  of  people,  equipment,  materials,  methods 
and  environment  that  produce  output  (product  or  service).  It  is  a 
repeatable  sequence  of  activities  with  measurable  inputs  and  outputs. 

Parameter:  A  measurable  characteristic  of  a  product  or  process. 

Process  Parameter:  A  measurable  characteristic  of  a  process  that  may 
impact  product  performance  but  may  not  be  measured  on  the  product. 
(The  “x.”) 

End-Product  Parameter:  A  parameter  that  characterizes  the  product  at 
the  finished  product  stage.  (The  “Big  Y.”) 

In-Process  Product  Parameter:  A  parameter  that  characterizes  the 
product  prior  to  the  finished  product  stage.  It  is  measured  on  the  product 
upstream  and  is  the  result  of  a  process  step.  (The  “little  y.”) 

Input  Variable:  An  output  from  other  processes.  (Neither  x’s  or  y’s.) 
[Hexsab  02] 
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Cause  &  Effect  Matrix:  Procedure 

Identify  the  y’s  from  process  map. 

Rate  the  y’s  on  a  scale  from  1  -1 0. 

•  Involve  the  “customers”  to  determine  the  ratings. 

•  Ratings  are  relative. 

List  the  process  steps  and  all  of  the  x’s  from  the  process  map. 

Rate  the  relationship  of  each  x  to  each  y  on  a  0,  1 , 3,  9  scale. 

0  =  No  relationship  between  x  and  y 
1  =  Remote  relationship  between  x  and  y 
3  =  Moderate  relationship  between  x  and  y 
9  =  Strong  relationship  between  x  and  y 

For  each  x 

•  Multiply  each  relationship  rating  by  the  corresponding  y  rating 

•  Sum  the  products 

Use  the  summations  to  rank  and  select  x’s  for  future  experiments  or 
focused  efforts 

[Hexsab  02] 
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Cause  &  Effect  Matrix:  Format 


Ys: 

Y  ratings: 

Process  steps 

Xs 

X  relationship  to  Y 

Sum 

[Hexsab  02] 
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Tool  Tips  Part  2:  Beyond  Basics 

Overview  (description,  procedure,  tips,  examples)  for 

•  capability 

•  voice  of  the  customer 

•  management  by  fact 

•  process  mapping 
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Tooltip:  Process  Capability 

Description 

•  When  a  process  is  in  statistical  control  with  respect  to 
a  given  set  of  attributes,  we  have  a  valid  basis  for 
predicting,  within  limits,  how  the  process  will  perform 
in  the  future. 

Usage 

•  Addresses  predictable  performance  of  a  process 
under  statistical  control. 

•  For  a  process  to  be  capable,  it  must  meet  two  criteria: 

-  The  process  must  be  brought  into  a  state  of 
statistical  control  for  a  period  of  time  sufficient  to 
detect  any  unusual  behavior. 

-  The  capability  of  the  process  must  meet  or  exceed 
the  specifications  that  have  to  be  satisfied  to  meet 
business  or  customer  requirements. 
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Histogram  Reflecting  Process 
Capability 


LCL=  36.08 


Voice  of  the  Process 

CL=  45.06 


UCU54.04 


Product  Service  Staff-Hours 
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Process  Capability  vs.  Capable 

k=i Voice  of  the  Process 


LCL=  36.08  CL=  45.06  UCL=54.04 
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Tool  Tip:  Voice  of  the  Client  (VOC) 

Description 

•  a  method  to  describe  the  stated  and  unstated  needs  or 
requirements  of  the  customer 

•  can  captured  in  a  variety  of  ways:  direct  discussion  or 
interviews,  surveys,  focus  groups,  customer 
specifications,  observation,  warranty  data,  field  reports, 
complaint  logs,  etc. 


[isixsigma] 
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VOC:  Usage 

Feeds  Quality  Function  Deployment  (QFD) 

Risks 

•  anecdotal,  not  quantitative 

•  difficult  to  get  “the  right  answer” 

•  humans  are  PERFECT  FILTERS! 

•  it  is  very  easy  to  induce  bias  in  VOC 

Tips 

•  use  existing  information  with  care  -  it  may  be  biased  or  too 
narrowly  focused 

•  always  use  more  than  one  source 

•  customer  visits  allow  direct  discussion  and  observation 

•  customer  visits  allow  immediate  follow-up  questions  and 
unexpected  lines  of  inquiry 
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VOC  Interviews:  Procedure  1 

•  Define  the  customer 

•  Select  customers  to  interview 

-  Always  interview  more  than  one 

•  Plan  interview 

-  Develop  a  checklist/guideline 

-  Teams  of  3:  “Moderator,”  “Scribe,”  “Observer” 

•  Conduct  interviews 

-  Customer  statements  &  observations  need  to  be 
recorded  VERBATIM 

-  Keep  asking  “why” 
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VOC  Interviews:  Procedure  2 

Create  VOC  table. 

•  Interpret  verbatim  statements  into  new  meanings. 

•  Document  source  of  VOC  or  re-worked  VOC. 

-  “I”  if  internally  changed  or  generated  (by  team) 

-  “E”  if  externally  generated  (by  customer)  or  not 
changed  by  team 

•  Classify  each  statement  as: 

-  a  real  need  l  feeds  QFD 

-  a  technical  solution 

-  a  feature  requirement  l  feeds  QFD 

-  not  a  true  need  (e.g.,  cost  issue,  complaint, 
technology,  hopes  dreams,  etc.) 

•  Quantify,  Analyze,  Prioritize 
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VOC:  Example  Table 

New  process  initiative  under  consideration 

•  interview  statements  recorded  verbatim  and  classified 

•  column  added  for  keyword  sorting 

Further  development 

•  “interpreted”  comments  about  the  organization’s  true 
goals,  the  overlap  of  initiatives  (and  so  on) 

•  evaluation  for  common  themes 

•  additional  data  collection  may  be  needed 


Classification 

Customer  comment 

Interpreted,  reworded 

l/E 

perception, 

experience,  context 

barrier 

root  issue? 

results,  success 

need 

solutions 

Keyword 
for  sorting 

We  are  already  at  maturity  level 
x,  so  why  do  more? 

E 

✓ 

competing 

initiatives 
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VOC:  Analysis 


Prioritization 

Method 

Customer 

Time 

Preparation 

Complexity 

Analysis 

Complexity 

Quality  of 
Resulting 
Prioritization 

Number  of 

customers 

needed 

Number  of 
Needs  to 
Prioritize 

Recommend 

Frequency  of 
Response 

short 

low 

low 

low 

large 

large 

NO 

Constant  Sum 

medium 

medium 

medium 

medium 

medium 

small 

Yes 

Rating 

short 

low 

low 

medium 

medium 

med-large 

Yes 

Simple 

Ranking 

medium 

low 

low 

medium 

medium 

small-med 

Yes 

Q-Sort 

short 

low 

low 

medium 

medium 

large 

Yes 

Paired 

Comparison 

long 

medium 

high 

high 

large 

small 

Yes 

Regression 

short 

medium 

high 

high 

large 

small-med 

Yes 
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Tool  Tip:  Management  by  Fact  (MBF) 

Description 

•  a  concise  summary  of  quantified  problem  statement, 
performance  history,  prioritized  root  causes  and 
corresponding  countermeasures  for  the  purpose  of 
data-driven  project  and  process  management 

Management  by  Fact 

•  uses  the  facts 

•  eliminates  bias 

•  tightly  couples  resources  and  effort  to  problem-solving 
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MBF:  Procedure 

Identify  and  select  problem 

•  use  “4  Whats”  to  help  quantify  the  problem  statement 

•  quantify  gap  between  actual  and  desired  performance 

Determine  root  cause 

•  separate  beliefs  from  facts 

•  use  “7  Basic  Tools” 

•  use  “5  Whys” 

Generate  potential  solutions  and  select  action  plan 

•  Must  be  measurable/sustainable 

•  Specific/assignable  ownership 

•  Understand  expected  results  from  each  action 

Implement  solutions  and  evaluate 

•  Compare  data  before  and  after  solution 

•  Document  actuals  and  side  effects 

•  Compare  with  desired  benchmark 
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MBF:  4  Whats 

Customer  satisfaction  scores  are  too  low. 

•  What  is  too  low? 

Compared  to  best-in-class  benchmark  of  81%,  we  are  at  63%. 

•  What  is  the  impact  of  this  gap? 

It  represents  lost  revenue  and  earnings  potential? 

•  What  is  the  correlation  between  customer  satisfaction  and 
revenue? 

Each  percent  of  customer  satisfaction  translates  to  0.25  percent 
of  market  share  which  equals  $100M  US  revenue. 

•  What  is  the  lost  potential? 

Final  problem  statement: 

Customer  satisfaction  is  18%  lower  than  best-in-class  benchmark, 
which  corresponds  to  a  potential  lost  revenue  of  $1 .8B  US. 
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MBF:  5  Whys 

The  marble  in  the  Jefferson  Memorial  was  deteriorating. 

Why? 

The  deterioration  was  due  to  frequent  cleanings  with  detergent. 
Why? 

The  detergent  was  used  to  clean  bird  droppings  from  local 
sparrows. 

Why? 

The  sparrows  were  attracted  by  spiders. 

Why? 

The  spiders  were  attracted  by  midges. 

Why? 

The  midges  were  attracted  by  the  lights. 

Solution:  Delay  turning  on  the  lights  until  later  at  dusk. 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


page  176 


Carnegie  Mellon 

Software  Engineering  Institute 


MBF:  Format 


FACTUAL  STATEMENT  OF  PROBLEM,  PERFORMANCE 
TRENDS  &  OBJECTIVES 


Prioritization  & 
Root  Cause 

Counter  Measures  & 
Activities 

Impact,  Capability 

List  of  gaps  in 
performance  and  true 
root  cause 

List  of  specific  actions,  who 
has  ownership  and  due  date 

List  of  expected 
benefits  and  impact  of 
each  countermeasure 
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Graph  of 
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MBF:  Example 

Problem  Statement _ 

Customers  A,  B  and  C,  representing  x%  of  market  share,  are  facing  budget/cost  constraints  and  require  a  10%  cost  reduction  in  our  product  line  XYZ  in 
order  to  continue  doing  business  with  us.  Baseline  data  shows  that  33%  of  software  development  time  is  spent  detecting  and  correcting  defects. 


200.0 
180.0 
0160.0 
9140.0 
*120.0 
ml  00.0 
o  80.0 

■S  60.0 
q  40.0 
20.0 
0.0 


Change  in  Defect  Density 


El 


<v-°  * 


□  Pre¬ 
improve 
ment 

■  Post- 
Im  prove 
ment 


Defect  Density 


Program 


Prioritization  &  Root  Cause 

Counter  Measures  &  Activities 

Who 

When 

Large  Quantity  of  Syntax  &  Similar  defects 
that  are  repaired  in  <10  minutes  on  avg 

Goal  is  50%  reduction  in  time,  relative  to 
historical  data 

Clarify  type  definitions 

jms 

S 4/30/2001 

Improve  subcategory  data  collection 

jms 

v' 4/30/2001 

Build  a  cause  &  effect  diagram  to  be  used  for  next  round  of 
analysis,  improvement  planning 

jms 

Increase  correction  efficiency  by  seeking  all  occurrences  of 
a  defect  upon  the  detection  of  the  first  occurrence 

jms 

v' 4/30/2001 

Increase  and  log  (new)  usage  of  off-line  programs  to  test 
small  pieces  of  functionality 

jms 

Create  &  Use  a  syntax  checklist 

jms 

S 4/30/2001 

"Big  Hitter"  (>10  minutes)  defects  involve 
a  variety  of  errors  that  escape  to  testing. 
Design-injected  and  Test-removed  errors 
fall  into  this  category 

Goal  is  25%  reduction  in  time,  relative  to 
historical  data 

Time  breaks:  phase  completion  &  every  hour 

jms 

V 4/30/2001 

Conduct  a  phase  check  prior  to  moving  on 

jms 

S 4/30/2001 

Increase  and  log  (new)  usage  of  off-line  programs  to  test 
small  pieces  of  functionality 

jms 

S 4/30/2001 

Improve  subcategory  data  collection  to  use  for  developing  a 
more  directed  design  review 

jms 

-4 4/30/2001 

Build  a  cause  &  effect  diagram  to  be  used  for  next  round  of 
analysis,  improvement  planning 

jms 
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Tool  Tip:  Process  Mapping 

Description 

•  representation  of  major  activities/tasks,  subprocesses, 
process  boundaries,  key  process  inputs,  and  outputs 


INPUTS 

(Sources  of 
Variation) 

•  People 

•  Material 

•  Equipment 

•  Policies 

•  Procedures 

•  Methods 

•  Environment 

•  Information 


OUTPUTS 

PROCESS  STEP 

(Measures  of 

A  blending  of 

Performance) 

inputs  to  achieve 

•  Perform  a  service 

the  desired 

•  Produce  a  Product 

outputs 

•  Complete  a  Task 
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Mapping:  Usage 

Feeds  other  tools 

•  Cause  and  Effects  Matrix 

•  Failure  Modes  and  Effects  Analysis  (FMEA) 

•  Control  Plan  Summary 

•  Design  of  Experiments  (DOE)  planning 

Tips  for  mapping  current  processes 

•  Go  to  the  actual  place  where  the  process  is  performed. 

•  Talk  to  the  actual  people  involved  in  the  process  and  get  the 
real  facts. 

•  Observe  and  chart  the  actual  process. 

•  Consider  creating  “as  is”  and  then  “to  be”  maps. 

Reality  is  invariably  different  from  perception  -  few  processes 
work  the  way  we  think  they  do! 
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Process  Mapping:  Basic  Procedure 

List  inputs  and  outputs 

Identify  all  steps  in  the  process:  value-added  and  non¬ 
value-added 

Show  key  outputs  at  each  step  (process  and  product) 

List  key  inputs  and  classify  process  inputs 

Add  the  operating  specifications  and  process  targets 
for  the  controllable  and  critical  inputs 
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Process  Mapping:  Example 

Inspection  process  from  earlier  illustration 


Plan 


O  Artifacts  to 
inspect 

#,  *  Artifact  size 

•  Reviewers 

*  Data  repository 


Detect 
Defects 

O  Review  Rate 
O  ,  *  Checklists 
#,  *  Inspection 
method,  procedure 
t  Proficiency 
f  Taxonomy 
interpretations 


Detect 

Defects 


Trouble¬ 

shoot 

What  would  you 
list? 

ODefect  attributes 

•  Proficiency 

•  Effort 

•  Tools 


Correct 

Defects 

What  would 
you  list? 

O  Correction 
action 

*  Config  control 
O  •,  Effort 


•  Defect  Log  •  Direct  Cause  •  Corrective 

•  Record  of  plan  •  Root  Cause  Action 


( - 

A 

Inspection 

O 

Critical  Inputs 

\ 

*  Standard  Procedure 

O 

v _ 

Rework 

t 

Noise 

•  Control  Knobs 

_ y 
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Process  Mapping:  Value  Map 

Identify  the  process  to  map 
Identify  the  boundaries 

Create  input-process-output  for  the  critical  processes 

Create  the  process  map 

Color  code  each  step  identifying  value 
green  =  value  added 
red  =  non  value  added 
yellow  =  non  value  added  but  necessary 

•  Identify  hand-off  points,  queues,  storage,  and  rework  loops 
in  the  process 

•  Quantitatively  measure  the  map  (throughput,  cycle  time,  and 
cost) 

•  Validate  map  with  process  owners 
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Value  Mapping:  Change  Request 


non  value  added 
non  value  added  but  required 
value  added 

Initial  Assessment  will: 

•  Determine  Impact  Assessment 

•  Identify  Stakeholder 

•  Coordinate  with  Product/Process  Owner 

•  Perform  Impact  Analysis 
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